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This study evaluated the implementation of and the' 
outcoiies from a local child parent center compensatory education 
program. It intended to contribute to a better understanding of • one 
phase of compensatory education; and, in the process, it proposed to 
develop a relatively simple and practical evaluation strategy which 
would verify the implementation of a program and relate the 
implementation measures to* the outcome measures. The evaluation 
strategy included practical methods for modeling the ptogram, 
gathering data, and anarlyzifeiplata. Causal modeling techniques were 
used to relate implementatifowweasures to outcome measures. The 
strategy was than analyzed regarding its usefulness as ah evaluation 
design which would measure program^ implementation and provide 
explanatory power. (Author/PN) ^ 
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Thil itudy evaluated the inplenentation of and the outcones froa a local 
child parent center conpeniatory education prbgraa. The evaluation ^ 
strategy included practical nethods for nodeling the prograSp gathering 
data, and analyzing data. Cauial modeling techniqi^s were used to 
relate inplenentation nei^sures to outcone measures. The strategy was 

0 

then analyzed regarding its usefulness an an evaluation design which 
would neasure progran inplenentation and provide explanatory power. 



In the t]rpicel queei-experiaentel evaluation, two Won'-equivalent 
groups, experiaental vs. control, are cospfred post treataenfe^ith j 
atjteapts nade to enaure pre-treataent equive^lence via analysia of 
covariancf (AHCOJA) . Lord lhas noted that, "i|[o logicial or statistical < 
procedure can ba counted on tp aake proper allowances for uncontrolled 
pre-existing differences between groups" (p. 305). Additionally, ANCOVA 
has been' found, typically, to underadjust for ^re-treat«ent differences 
(Campbell & Boruch, 1975); thereby not sinply providing equivocal 
results, but equivocal results which indicate, that treatment was 
hatful . 

Campbell and Boruch suggest that this situation bf remedied by the 
greater use of experimental designs (random assignment). Additionallly, 
Hagidson and Sorbom (1992) have demonstrated that structural equation 
modeling techiyqu^s allow more precise estimates of pre-treatment 
differences between groups, i.e., the true score differences between 
groups. This technique>equires multiple measures of independent and 
^ dependent variables. The problems with these two remedies for 
quasi-experimental designs are that (1) usually it is difficult to 
implement "and maintain random assignmeii^, and (2) that local avaluators 
often lack the wherewithal to obtain multip^ measures and perform the 
I intricate statistical analyses required by structural equation nbdels. ' 
Are there any more alternatives? Recent research suggtfTts that the 
emphasis' on outcomes, while especially important in traditional 
psychological research, is somewhat off-target in evaluation studies 
(Charters i. Joqes, 1973; Efikson, 1979; Fullan Pomfret, 1977; j 
Leinhardt, 1980). The study at hand finds Chat an em^asis on measuring 
program processes (implementation) sh^ds light on the question of 



prograa valu« by first ^aiking not **Did th« •xperiamt work?" but "Wai 
th«r« an expariMnt?** In oth«r words, prograas which ar« foimd to 
successfully ii^Uaont tho intondod troataont ara inharantly «ora ^ 
valuable and, other things being equal, have a theoretically better 

chance of causing the intended outco«eSo« • 

ft t 
The Need for Process Measures' 

In spite of the growing awareness of the need to "study processes in 

order to account fof outcones, actual evaluations of school progress 

have typically continued to concentrate on ^u^cones (usually achievenent 

alone). Boruch's recent reconendations to Congress advocating the 

routine neasureaent of iapleaentation (Boruch, Cordray, Pion, li Leviton, 

1981; Boruch fc Cordray, 1980) were founded on recent research such as 

that conducted by Forgione, Kaplan, and Orland (Note 1) in "Evaluation 

of Coj^pensatory £ducation Programs: Probleas, Proaising Strategies and 

Recent Trends.** The authors studied the types of evaluation strategies 

enployed in evaluating Title I prograas by S2 school districts in eight 

states. , They found tl^at evaluation priaarily aeant achieveaent testing 

to both states and local districts and that evaluation was viewed aost 

frequently as a mechanical exercise unrelated to other adainistrative 

activities. It was |purther found, by David (Note 2) in her study of 

evaluation utilization at the local level that local evaluations were 

not used either as a basis for judging the effectiveness of Title I 

prograas or as guides for program decision-making.^ 

With the development of improved methods of measuring program 

implementation (Leithwood li Montgomery, 1982; Rubin, Stuck, li Revicki, 

1982; Gersten, CatKine, li Williams, 1982; DeVault, Harnishfeger, It 

Wiley, 1978) there arises the need for practical aethods of utilizing 
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thos« inplMtntation measurts in local prograa avaluations. The prasant 
study daaonstratas ona such sathod using basic causal aodaling 
techniques to relate neasuras of prograa iapleaentatioiv to aeasurea of 

A ^ 

prograa^'outcosa. ^ ^ ^ ' 

General Objectives 

The general purpose of this study was to verify the iapleaentation 
of the intended treatments in the Chicago Child Parent Center (CPC)^ 
compensatory education prograa (Conrad fc Eash, 1982). With this general 
purpose, the study intended to contribute to a better understanding of 
one phase of compensatory education; and, in the process, it proposed to 
develop a relatively sitaple and practical evaluation strategy which 
would verify the iaplementation of a prograa and relate the 
iaplementation measures to the outcome measures. 

It should be noted that the evaluation strategy proposed here did 
not propose to offer solutions to the much discussed problem of 
measurement error (e.g., Borucb Gomez, 1979). Rather, it simply 
attempted to apply recent advances in evaluation technology employing 
basic causal modeling techniques to a local program evaluation. 
Compensatory Education Programs 

Compensatory education programs have, of course^ been studied 
before; most notably, the Head Start program. The negative findings 
from the evaluation of Head Start began a continuing discussion of 
methodology in evaluating large social programs (Westinghouse/Ohio 
University, 1969). Many problems flawed the Head Start evaluation, but 
the main problems of concern here are that Head Start treatment was 
poorly specified and that implementation of treatment was poorly 
verified. The evaluation strategy described here addressed these two 



9onc«rnf l^y d«lin«atinf th« ittps needed to specify and verify th« 
iaplraentftti6n of trMtsmt. 

Methodi 

With thii bri«£ background, this papar turns to a description and 
deaonstration 0f an •valuati'Vi strategy which enployad iaplaaentation 
neasures and related thea to outcoae neasures with the purpose of aiding 
causal inference. The data used in the denonstration of the «odel were 
taken fron the Chicago Child Parent Center Study (Conrad & Eash, 1982). 

A causal pf^ograa aodel was constructed. The model was based the 
intentions of the program administrators as to the processes and 
outcomes which pade up tke program. The use of a model greatly 
facilitated our ability to think about and analyze program 
implementation and Outcomes. ** 

Specif ideally, the CPC model for improving academic Jchievement 
assumed that criterion performance of students on achievement tests was 
affected by the following classroom variables: 

— A reduced class size, which was intended to allow 

(J 

increased individualization and increased interaction with 
the teacher and teacher-aide, thus producing a 
more child-centered classroom J/ . 
. ^^A highly structured basic skills program with increased 
student evaluation 
— An Abundant supply of learning materials resulting in 
an enriched educational environment 

( 

— Parent involvement in the school program • 
Figure 1 depicts a theoretical model of the CPC program. 
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Insert Fifur* 1 about h«r«* 



Appropriatt initrwnti to irfaiurt proc#ii#i and outcoaai 
were aaUctad> In tha CPC itudy, tha "Clasirooa Obiarvation Rating 
Scala" (Walbarg and Thosas, 1974) was adaptad to aifaff clasirooa 
traataant. It was baliavad that t^a CORS was appropriata for this 
avaluation becausa tha* theaas covarad by tha CORS wara siailar to tha 

A, 

intandad procaisas of tha CPC prograa; Tha CORS was davalopad following 
tha coapansatory education aovaaant and aaphasizad thaaas which wara 
siailar to thosa intandad by tha philosophy of coapaifsatory education. 
Regarding the reliability of the CORS, H»lberg and Thoaas si«^ 
Statistical analysis of the resulting data showed that 
the total score on the scales is highly reliable with 
internal consistencies above .95 whether obtained through 
questionaire or observation. Moreover, they discriainate 



British schools in advantaged and disadvantaged areas at 
high levels of statistical significance (1974). 
Three scales were delineated a priori froa tho CORS: 1) 
child-centeredness; 2) presence of evaluation of student achieveaent; 3) 
enriched environment; and a fourth scale, parent involvenent, was 
constructed specifically for this evaluation (see Table 1). 



between open and traditional classes in Aaeri/can and 




Insert Table 1 about here. 
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Th« outcos« aeafur* uf«d was the Prereading AchieveMlit Scale of tha 
Coaprahaniiva Taiti of Baiic Skilli , Laval A, Pona S (CTBS, 1973), CTBS 
raportf intarnal consistancy caliabilitiai of .9^ for tha Total 
Praraading Scala, 

A cadra of trained obiarvari wai obtainad. In the CPC study a cadre 
of fcbool and conunity rapresentativai froa participating public 
schools and faculty and graduate assistants froa the university were 
trained to collect| data. A singl^e, one hour training session was held 
in which items waire read and axaaples of situations which wouldl apply to 
particular iteils were presented. The faculty and graduate assistants 
who participated in the i;lassrooa observations had previously worked on 
siailar projects were cd/nsidered experienced and expert observers. 

Measures of classrooa processes wate obtained froa a randoa 
sample of 13 treataent and 6 control classes. The sample population of 
the CPC implementation s^udy was composed of classrooms of students from 
age cycle 5 in four CPC*s fnd affiliated public schools as well as from 
two schools not in any way Affiliated with CPC*s. All schoola were 
located in the same school dii^trict and Served similar disadvantaged 
populations of whom the overwhelming majority was black. ) 

The major problem with this iptudy was the small sample of classrooms 
and the consequent low power to detect effects. The reader should bear 
this in mind when effect sizes are reported as non-^signif icant at p<.05. ' 
On tha other hand, tha fact that ^lass means were used on the dependent 

measure made this estimate relatively stable. 

/ . 

Class means on measures of atudfnt outcomes were obtained 
from the sample classrooms. It was necessary to obtain class means 
because the classroom was the appropriate unit of analysis. In this 
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.analysis, tha S-^aar-old students wara si»rtad by rooBr Md thair 
Pi^^r^ading Achiaveaant s cor as wara avaragad. 

Rasults 

Oiitcoaas wara related to treatment to datanoina tha strength 
of the relationship . This step alone is the crucial analysis of the 
typical outcomes evaluation. % 



Insert Figure 2 about here. 



Figure 2 reveals that when Prereading Achievement was re|ressed on CPC 
Program (dummy coded: CPC-1, non-CPC«0) the path coefficient was 
non-signif iitant. In tha typical outcoiies evaluation this would indicate 
that the program had no affect. However, the following path models 
demonstrated Chat a great deal of information about how the CPC Program 
affected classroom processes and about how classroom processes affected 
Prereading achievement was absent from this analysis. 

Htf magnitude of the process measures was observed to determine 
whether they .were being implemented at intended levels. In the CPC 
implementation study, it was decided that a mean item response above 2.5 
on a A point scale would be considered tftrong evidence and below 2.5 
weak evidence of the presence of the yarious classroom processes. 2.5 ^ 
was chosen simply because it was tha mxd-point on the 4 point scale. 



Insert fible ¥ about here. 



Table 2 reveals that tha CPC classrooms scored above 2.5 on the 
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Evaluation and Enrichad Environmant scalat, indicating strong 
iaplaBantation of thaaa classrooa variablaa.' Howavar, / 
Child*cantartdnaaa was found to ba balow 2.5 but was significantly 
highar (p<.05) than non-CPC classrooas. In this casa, tha critarion of 
statist icaf significapca was thought «ora coapalling and valid than tha 
arbitrary critar^ 6f tha 2.5 nid-point, and this finding was 
intarpretad as'^aing significant. Thus, although both CPC and non-CPC 

classrooms wara found to ba low in Child-cantaradnass* CPC classrooas 

} • * 

wara intarpratad as baing significantly highar in this attributa. 
Parant Involvaaant was balow 2.5 and showad no jitatisticaily significant 
diffaranca batwaan CPC and noif-CPC classrooas. 

Procassas wara ralatad to traataan^ to datanaina if tha intandad 
prograa procassas wara actually baing iaplaaantad to a significantly 
graatar dagraa than in convantional projtraas. Thia was tha "program 
tasting staga*' of tha analysis. In othar wor^s, this analysis tastad 
whathar tha axpariaantiil prograa was a significant iaprovaaant ovar tha 
convantional prograa on aaasuras of tha spacifiad variablas. In tha CPC 
study tha procasa aaasura (CORS Total) was ragrassad on traatmant (duaay 
codad 1-CPC, 0-non-CPC) . Tha ragrassion astiaatad tha strangth of tha 
ralationship bal^waan tha procas^ aaasuras and baing a CPC classrooa. 
Tha laft sida of Figura 3 dapicts tha strangth of tha ovarall ragrassion 
of fraataant on Procass Total, tha sua of tha four fcal as of tha CORS. 
Figux'a 4 dapicts tha rasults of tha ragrassions of saparata procass 
^scalas on Traatmant. This modal involved four saparata aquations, tha 
regressions of each scale of tha CORS on traatmant. The path 
coefficients on tha left in Figure 4 reveal that CPC treatment caused 
statistically significant differences in "child-centeredness" and 
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"evaluation" but not in "•nrichod onvironaont" and "parent involvoaont." 

t 

In««rt FifurM 3 and 4 about hara. 



Outcoaai wara ralatad to procaiiai to datamina tha itrangth 
of th^ raUtionihip, Thii was tha *'thaory tatting (taga" of tha^ 
analyiii. In othor wordi, thii analyiii toitod whothor tho intondod CPC 
proceiiei war* lignificantly ralatad to outcoaai. If thara wara not a 
strong ralationship batwaan outcoaai and procaiiai, it would ba 
inconsistant with tha infaranca that tha procaiiaa causad tha outcoaas. 
In this daaonstration, only ona outcoat was usad (i.a., Praraading 
Total) but classrooa procassas wara analyzad both individually and as a 
whola. 

Tha right sida of Fi^ra 3 illustratas that tha ragrassion of 
Praraading Achiavaaant on tha total of tha procass aaasuras was 
statistically significant, indicating that ovarall tha procasa aaasuras 
wara ralatad to tha outcoaas. Howavar, tha right sida of Pigura 4 
illustratas that whan tha individual scalas wara antarad in a 
hiararchical ragrassion analysis (i.a., Child-Cantaradnass first. 
Evaluation sacond, Enrichad Environaant third, and Paradi Involvaaant 
fourth), nona was fpund to hava a significant ralationship with 
Praraading Achiavaaant. This was intarpratad to aaan that whan a total 
prograa including all four of thasa alamants is iaplaaantad, it doas 
hava tha dasirad affact of iaproving achiavaaant. In othar words, in 
this study, tha individual alaaants did not saaa to contributa^ 
significantly to tha pradiction of Prareading Achiavaaant, but tha 

coabination of thaa did. ^ 

% 
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^ Dilcusiion 

In this case, the inplementetion of the CPC progren was supported 
insofar as it was found that the CPC program was strongly related to two 
of the intended processes and that theae procesns, taken as a whole, 
were strongly related to increased achievenent. This finding belies the 
fact that the CPC program (when employed as the. sole. independent 
variable) did not show a statistically significant gain in outcomes. 

Suchman (1967) and Shapiro (1982) have distinguished between program 
failure (i.e., the program is not implemented as intended) and theory 
failure (i.e., the program is implemented as intended but does not cause 
the expected results). In this case, the program was found to be 
implemen|ed'as intended for two variables but npt for two others, 
thereby indicating some program success. Additionally, the processes, 
taken as a whole, were found to be significantly related to outcomes, 
indicating somre theory success. Therefore, it may logicall^be inferred 



that an intensification of Enriched Environment and Parent Involvement 
should cause an increase in PArereading Achievement. It was concluded ~ 
that the theory of the program was, in general, correct, and that the 
program was well implemented in terms of Chi Id-Center edness and 
Evaluation. If there was a failure in the program, it was not due to 
failure of the causal model, but to the fact that the differences in 
implementation of the intended program were not great enough between the 
experimental and* comparison programs to provide statistically 
significant outcomes in the simple rejj^ression analysis. To improve the 
program, the environment should further enriched relative to the 
regular program and parent ihvj^lvement should be increate4« ^ 
Finally, it should be recalled that this study was viewed \s a 



prototype useful for the develbpment of an evaluation design not at a 
crucial experiment. In fact, the study from which these data came had ^ 
multiple indications of the efficacy of the CPC Program. A program 
rai/ely intends only one outcome, so multiple measures 9re necessary to 
ensure that the program addresses the interests of a diverse policy 
shaping coipnunity. ' 

^ ^ Conclusion . 
In conclusion, this study supported the strategy employed in the 
Chicago Child Parent Center Program for remediating student achievement. 
In addition, the program evaluation design which was described here is 
thought to be a significant improvement in evaluation technology 
becauae: 

1. It measures implementation of the program as well as outcomes, 

•= • . . 

thereby providing a basis for causal 'inferences . 

2. It provides a method for specifying the theofy of the program and 
a method for testing that theory. 

3. It employs available personnel efficiently. 

A. It calls upon the available expertise of staff researchers, teachers 

■ i 

school and community representatives and others. 
5. It uses much available data productively (e.g., achievement date). 
.6. It is minimally disruptive of school dnd classroom processes. 

7. With careful sampling, it can provide valid* results without an 
extensive and costly data collection. 

8. While providing data useful for decision-making, it also provides, 
data useful for program development and improvement. 

Using program implementation measures in path analysis of causal 
models provides information about the "inner workings" of the program. 



Although the problem of ibeasurenent error is a constant barrier to 
drawing causal inferences » the moling process described here provides 
substantial benefits in verifying the implementation of treatment and in 
testing program implementation and program theory as well as program 
outcomes. 
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Th« Four A Priori Scalei of th* "Claiirooa Obiarvation Ratint Scala" 
(Walb«rt Thomas, 1974) and Their Itcfa Rapresahtation 



Sample Itemi ^ N of Items 



1. Child'centeredness: 

Children work individually and in small groups 
at various activities* Children are expected 
to do their own work without getting help from 
other children (R).* Teacher bases instruction 
oil each individual child and his/her interaction 
with materials and equipment. 



21 



Measures 

Implementation 
of 



Reduced class size* 

individual izat ion* 
humaneness « 



2. Evaluation: 

Teacher gives childr.en tests to find out what 
they know. Teacher views evaluation as inform- 
ation to guide instruction and, previa ioning for 
the classroom. ^ 



Structured basic 
skills program 
and evaluation 



3* Enriched Environment: 

Materials are readily accessible to children. 
Books are supplied in diversity and profusion 
(including reference* children's literature)* 
Manipulative materials are supplied in great 
diversity and range* with little replication* 
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Provisioning for 
abundant materials 



4. Parent Involvement: 



There are parents in the classroom, 
The environment includes materials fpr 
parents to read or use. ^ 



Parent involvement 
in the classroom 



* The i*R*^ means 'that coding was reversed when this item was scored. 



* • 
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Table 2 

Group Means, Standard Deviations, ^nd Mean Ite« Values of .CPC ys. non-CPC 

Classrooas on the Four A Priori Scales of the CORE «, ^<^' 

Means Standard Meinltea Valiie* 

Deviations (4 ^int scale) ' 

■ 

Scale CPC non-CPC CPC non-CPC CPC non-CPC 

Child- " 

centeredness 45.4* 32.0 6.3 4.1 

Evaluation |^ 18.0** 8.5 5.4 2.0 

Enriched 

Environment 29.2 26.0 4.9 5.2 

Parent ' / 

Involvemfnt 8.2 7.1 2.7^ 2.7 

m CPC n-13, non-CPC n"6 



b To obtain conperpbility of aeaiureaent on theie scelei, the group 
means were dividWi| by. the nuAer of items. This gave an indicati^ 
of relative magnitude on the four point scale. 

♦ p<.05 * 
p<.01 
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Figure 1. A theoretical causal model of the Child Parent Center Program. 
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Figure 2. A summative evaluation. 
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Figure 3. A conibination of formative and suinnative evaluation strategies. 
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PROGRAM TESTING STAGE 

Does the program have more of 
the intended program processes? 



THEORSr TESTING STAGE 

Are the intended program processes 
related to the desired outcome? 



CPC Program 




CORS Itotal 


.53* ^ 


• 

^Prereading 












Achievement 


i 



*p<.05 
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Figure 4. An exaxninntion of the strength of the relationships of 
individual processes ^d treatment and outcotnef. 



PROGRAM TESTING STAGE 

Each process scale was regressed, 
individually; pn treatment. 



CPC Program 
CPQ»1, non'^O 



THEOI^ 



TESTING STAGE 



Prereading achievement was regressed 
on the process measures via a^ hierarchical 
solution with scales having the strongest 
relationship with treatment given priority. 




Child^ 

centeredness 
via COTS 



Evaluation 
via C0R5 



Enriched 
Environment 
via CORS 



Parent 
Involvement 
via CORS 




Prereading 
Achievement 
via eras 



* P<.OS 
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